InfoMagic Internet Tools 1993 July

home *** CD-ROM | disk | FTP | other *** search

/ InfoMagic Internet Tools 1993 July / Internet Tools.iso / RockRidge / mail / pine / imap_archive / text0061.txt < prev next >

Wrap

Text File | 1993-07-02 | 25.7 KB | 646 lines

I am currently working on implementing the IMSP spec sent to this list earlier. Note that we are focusing on the needs of a large site with multiple IMAP servers, mobile users, and flexible administrative needs. I'm including a document I wrote outlining the implementation plan I am using. This is a fluid document which can change as the IMSP spec is updated, and as I encounter unexpected problems. I'm sending it so that you can get a better idea of how we're thinking and what we're doing at CMU. Comments and suggestions are welcome. - Chris Newman Carnegie Mellon University Computing Services --------------------------------------- *DRAFT*DRAFT*DRAFT*DRAFT*DRAFT*DRAFT*DRAFT*DRAFT*DRAFT*DRAFT*DRAFT*DRAFT* CMU's IMSP server implementation -------------------------------- by Chris Newman draft 3/26/93 ASSUMPTIONS ----------- This document assumes you are familiar with both the IMSP and IMAP2bis protocols. TERMS ----- CMU-IMSP CMU's implementation of IMSP IMAP Interactive Mail Access Protocol (see RFC-1176, IMAP2bis documents) A protocol for users to access mail and bboards. IMSP Interactive Mail Support Protocol (see IMSP document) A protocol to manage multiple IMAP servers and provide support functions that are related to mail access. Update string An implementation dependent string which uniquely specifies a message within a folder. USP Un-Specified Protocol A currently unspecified protocol or set of protocol extensions for communication between IMAP and IMSP servers. FILES ----- CMU-IMSP will store its configuration files in a configuration directory (usually /var/imsp). Within this configuration directory are the following files (the .lock files are empty files used to prevent synchronization errors): options, options.lock Global options file. See the "OPTIONS" section below. bboards, bboards.lock List of available bboards, the servers they're on and the update string of the last message in the folder. See "BBOARDS" below. changed, changed.lock List of bboards that have been renamed or aliased. user Directory for user specific information. The user subdirectory will contain another subdirectory for each user on the system. The user specific subdirectories will contain the following: options, options.lock User options. See "OPTIONS" section below. mailboxes, mailboxes.lock List of mailboxes the user has, a flag indicating subscription status, an update string of the last message for which the user has read everything previous, and the server they are on. See "USER MAILBOX LISTS" below. bbsubs, bbsubs.lock List of bboards the user has accessed, a flag indicating subscription status, and an update string. See "SUBSCRIPTIONS" below. When the CMU-IMSP server becomes a replicated service, cross server locking and synchronization of these files will need to be implemented. All file access and file locking will be heavily modularized in expectation of this replication. OPTIONS ------- Server configuration, user configuration and some general information is made available through the options interface. Options may be read with the IMSP "GET" command and changed with the IMSP "SET" or "UNSET" commands. Four basic types of options are supported. "Magic" options are built into the server and may return different values at different times (e.g. the "date" option). "Non-visible" options may only be used by the config administrator, and are for configuration options that are of no interest to the user. "Read-only" options may not be changed by users. "Read-write" options may have a default (global) value, which may be overridden by the user's local options. Non-visible options appear read-only to a full administrator. Magic options usually appear as read-only. The options file contains a list of options in the following format: OPTION-NAME OPTION-TYPE OPTION-VALUE LF The OPTION-NAME is a string containing no spaces or CRLF characters and specifies the option. The OPTION-TYPE is a single character either 'N' (Non-visible), 'R' (Read-only) or 'W' (Read-Write). The OPTION-VALUE field has the option value with '\n' for newlines, and '\\' for '\'. Option names are case-insensitive, but option values may be case sensitive. By convention: Boolean options are on if their value is '+' and off if their value is '-'. Integer options are numeric with an optional '-' prefix. List options begin with '(', end with ')', and the different items are separated by spaces. If necessary, '"' could be used to quote list items containing parentheses and spaces. The "imsp.options.quota" option specifies the maximum number of kilobytes permitted in a user options file. Normal users may only set options in their user options file that don't shadow a global read-only option. Full administrators may change global options using the IMSP "SET" command and prefixing the option name with a '*'. ADMINISTRATION -------------- There will be three levels of administration provided by CMU-IMSP. The usernames in the "imsp.admin.subs" list option are allowed to view but not change other user's subscriptions and mailboxes (by issuing the LOGIN command with a null password). This is provided to allow a subscription statistics service such as CMU's "arbitron" or a delivery system to find the location of a user's INBOX. The "imsp.admin.bboards" level would be useful for a postmaster who administrates the entire bboard tree. The "imsp.admin.all" level (which includes all the others) allows full access to the CMU-IMSP server and would be useful for the system administrator. The administrator options will be lists of userids. PTS or DFS groups may be also be supported by a compile-time option. The following IMSP commands will be generally restricted in CMU-IMSP: MOVE, MOVE.BBOARD, ALIAS.BBOARD, RENAME.BBOARD, CREATE or CREATE.BBOARD with server_partition_list, and DELETE or DELETE.BBOARD with hostname. If the bboard "user.<username>.INBOX" is removed, then that user will be removed from the IMSP server database. LOGIN ACCESS ------------ Access to the server is controlled by the login command. The initial version will allow kerberos-style logins as well as plaintext logins. If the global option "imsp.create.new.users" is not set, then a user must also have a subdirectory in the "user" directory in order to log in. If "imsp.create.new.users" is set and the user has no INBOX, it will be created according the the "CREATE" policy below. An administrator with appropriate access may gain access to a user's MAILBOX namespace by issuing a second LOGIN command with that user's name and a NULL password. SERVER STRUCTURE ---------------- The CMU-IMSP server will run as a process which watches the appropriate port. When a connection from a client is made, the server process will fork to give each client its own process. The parent process will continue to watch for connections and will also periodically update the BBOARD LISTS (see next section). A limit on the number of connections to service could be added if deemed necessary. FAULT TOLERANCE --------------- Any bad protocol or improper syntax from the user will be rejected through the protocol. If a connection from a user is dropped, the server will make sure all files are up to date, and let the connection go. In the case of server disk errors, the server will abort (and remove itself from the pool if replicated servers are being used) and wait for the system administrator to clean up. User subscriptions and options should be backed up regularly for recovery purposes. If a proxy connection to an IMAP server fails, IMSP will respond to the user request which prompted the proxy with a failure message. CONTACTING IMAP SERVERS ----------------------- Because the MAILBOXES namespace is user specific, CMU-IMSP must be able to do proxy logins to the IMAP servers. When a user logs into CMU-IMSP with a plaintext password, this is not a problem. However, when a user logs into CMU-IMSP with kerberos, CMU-IMSP can't pretend to be that user to the IMAP servers. In order to support kerberos, the IMAP server must allow the CMU-IMSP identity kerberos authenticator as a valid password for any user. Initially the CMU-IMSP identity will be "imap.*". When we implement our own IMAP server, it will keep a list of valid IMSP identities. BBOARD LISTS ------------ The first time the CMU-IMSP server is started, no top level "bboards" file will exist. At this point the server will check the "imap.servers" option and contact each IMAP server in the list to find the available bboards by doing a "FIND ALL.BBOARDS *". The bboards file will have the following format: BBOARD-NAME IMAP-SERVER-LIST UPDATE-STRING LF The BBOARD-NAME is the name of that bboard. IMAP-SERVER-LIST is a list of hostnames where that bboard is stored. If IMAP-SERVER-LIST is the empty list "()", then it is assumed that BBOARD-NAME is located on the same server as its parent bboard (using '.' notation). UPDATE-STRING is the update string for that bboard (optional). BBOARD NAMESPACE ---------------- If the "imsp.share.mailboxes" option is set, then the "user.<username>." namespace is reserved as a mapping from the MAILBOX namespace to the BBOARDS namespace. If the "imsp.external.subs" option is set, than names of the form {hostname}folder are reserved for external sites. USER MAILBOX LISTS ------------------ Information about the location of user mailboxes are stored in that user's "mailboxes" file. The structure of the file is as follows: MAILBOX-NAME SUBSCRIPTION-STATUS UPDATE-STRING [SERVER-NAME] LF MAILBOX-NAME is the name of the mailbox. SUBSCRIPTION-STATUS is a flag: '0' indicates user is not subscribed and '1' indicates user is subscribed. UPDATE-STRING indicates how much the user has seen. The optional SERVER-NAME specifies the hostname where the mailbox is located. If it is omitted, the server is assumed to be the one specified by the "imap.default.server" option. If user logs in and has no mailboxes file, CMU-IMSP will contact the IMAP server specified by the "imap.default.server" option, do a proxy-login and a "FIND MAILBOXES *". The "SUBSCRIBE MAILBOX" and "UNSUBSCRIBE MAILBOX" IMSP commands are used to modify the SUBSCRIPTION-STATUS field. If the option "imsp.external.subs" is set, then a user may subscribe to any mailbox name with the '{' prefix. The subscription will be returned by the FIND MAILBOXES or FIND UNSEEN.MAILBOXES with an empty list of server locations. This is only appropriate for sites whose clients all recognize the {hostname}mailbox notation. SUBSCRIPTIONS ------------- Information about subscriptions to mailboxes are stored in the mailboxes file (see the USER MAILBOX LISTS above). Information about subscriptions to bboards are stored in the user's "bbsubs" file with the following format: BBOARD-NAME SUBSCRIPTION-STATUS UPDATE-STRING LF BBOARD-NAME is the name of the bboard. SUBSCRIPTION-STATUS is a flag: '0' indicates user is not subscripted and '1' indicates user is subscribed. UPDATE-STRING indicates how much the user has seen. Subscriptions may be adjusted with the IMSP "SUBSCRIBE" and "UNSUBSCRIBE" commands. A user is not permitted to unsubscribe to a bboard listed in the "imsp.required.bbsubs" option. If a user has no bbsubs file, a new one will be created with a subscription to each bboard listed in the "imsp.default.bbsubs" option. If the option "imsp.external.subs" is set, then a user may subscribe to any bboard name with the '{' prefix. The subscription will be returned by the FIND BBOARDS or FIND UNSEEN.BBOARDS with an empty list of server locations. This is only appropriate for sites whose clients all recognize the {hostname}bboard notation. If a user tries to subscribe to a name of the form "user.<username>.*" and the "imsp.share.mailboxes" option is set, then the ACL for that folder (if existent) will be checked to see if the user has lookup access, and the subscription will be permitted if the user does. Error messages should not allow the current user to find out what folders <username> may or may not have. The mailboxes and bbsubs files will be kept in alphabetic order. By default, the FIND commands will return results in this order. A user may adjust the order returned by the "FIND BBOARDS *" and "FIND UNSEEN.BBOARDS *" commands by setting their "bboard.order" option. This option is a list of patterns (as would be used with the FIND command). For each pattern, all subscriptions that match that pattern (and have not yet been sent) will be sent in alphabetic order. Any subscriptions not matched by any pattern will be sent afterwards. The "mailbox.order" option functions the same, but it adjusts the responses to the "FIND MAILBOXES *" and "FIND UNSEEN.MAILBOXES *" commands. FIND UNSEEN ----------- The "FIND UNSEEN.*" commands will be implemented as follows: all new subscriptions will default to an unseen state, so by default "FIND UNSEEN.BBOARDS/UNSEEN.MAILBOXES" will be the same as "FIND BBOARDS/MAILBOXES". The "LAST" and "SEEN" IMSP extension commands will be used to change seen/unseen information as follows: tag LAST MAILBOX mailbox update-string userid tag LAST BBOARD bboard update-string These commands will be sent from an IMAP server to CMU-IMSP periodically to indicate the mailboxes and bboards with new messages. tag SEEN MAILBOX mailbox update-string userid tag SEEN BBOARD bboard update-string userid These commands will be sent from an IMAP server to CMU-IMSP only when a user finishes reading all messages on the specified mailbox or bboard. For bboards, a "LAST" will specify the update string to be placed the the bboards file for a given bboard, and a "SEEN" will specify the update string to be placed in the bbsubs file for a given user and bboard. For mailboxes, a "LAST" will specify that a NULL should be placed in the update string for that mailbox, and a "SEEN" will specify that that value should be placed in the update string. It is expected that the IMAP servers will be the only users allowed to use the "SEEN" and "LAST" commands. The "FIND UNSEEN.BBOARDS" command will list all bboards where the bbsubs update string differs from the bboards update string (or where the strings are NULL). The "FIND UNSEEN.MAILBOXES" will list all mailboxes with a NULL update string. IMSP SERVER REPLICATION ----------------------- A single IMSP server will probably be insufficient for a medium to large site. Therefore consideration must be taken on how to replicate the CMU-IMSP database between cooperating CMU-IMSP servers. An inter-IMSP server locking and data transfer protocol will need to be found. One possibility is to use the ubik protocol from Transarc. This would, however, prevent us from being able to distribute a replicated IMSP implementation outside of CMU. Load balancing between IMSP servers should be provided by DNS. For the first implementation (which won't include IMSP server replication), care will be taken to keep all access to potentially shared data highly modularized. BBOARD REPLICATION ------------------ IMSP supports replication of mailboxes and bboards on multiple servers. CMU-IMSP will only support replication of bboards. To do this, CMU-IMSP will have to designate a master site for a bboard and manage replication through the USP. The master site will be the first bboard listed in the server list. In addition, the USP may support a server-load indicator so that CMU-IMSP can sort the output of the list of servers for FIND command by a load parameter. Alternatively, the CMU-IMSP server could simply randomize the list. The FIND command will return unsorted server lists to any administrator (so that the administrator can determine the master site). If a "CREATE.BBOARD" or "MOVE.BBOARD" command with replication sites is specified and the bboard already exists on some of the replication sites, they will simply be added to the list and no error will be generated. MOVE ---- The IMSP "MOVE" and "MOVE.BBOARD" are used to move folders between IMAP servers. This requires a command in USP (or IMAP) to direct an IMAP server to send a folder to another IMAP server. CREATE ------ The IMSP "CREATE" and "CREATE.BBOARD" commands are used to create new mailboxes and bboards. Both commands adjust the mailboxes and bboards files as appropriate. If no hostname is specified on the "CREATE" command, the hostname will be selected as follows: 1) If the user has an INBOX, the server the INBOX is on. 2) If the user has a "imap.default.server" option set, that server is used. 3) If the USP has a "request server free space" option, then the server with the most free space in the "imap.new.mailbox.servers" list will be used. 4) Otherwise a random server from the "imap.new.mailbox.servers" list will be selected. "CREATE" will be implemented by proxy to an IMAP server. If no hostname is specified on the "CREATE.BBOARD" command, then the policy used to select a location for the bboard is specified by the "imsp.create.policy" option. Policy types to be implemented in CMU-IMSP include the following: "random" selects a random server from the "imap.new.bboard.servers" list. "parent" selects the server that the "parent" bboard is located on, if no parent bboard is found, it falls back to another policy. "free-space" selects the server with the most available free space. This can only be implemented if a free space request is available through USP. The "CREATE.BBOARD" command requires USP. CHANGED SUBSCRIPTIONS --------------------- The "changed" file allows a lazy evaluation method of updating user bbsubs files. When a user is subscribed to a non-existent bboard, the "changed" file will be checked for an entry for that bboard. The changed file contains lines of the form: OLD-BBOARD NEW-BBOARD TIMESTAMP LF OLD-BBOARD is the name of a non-existent bboard and NEW-BBOARD is the name of the new bboard which has replaced OLD-BBOARD. The TIMESTAMP is provided as a mechanism to expire changed entries to prevent the file from growing without bound. If the TIMESTAMP has a value of 0, then the entry represents a permanent alias. If a bboard is renamed more than once, the previous entries in the "changed" file will be updated to prevent chaining. "user.<username>.*" bboards will only be entered into this file if the "RENAME.BBOARD" command is used on them. Otherwise users are expected to keep track of shared mailboxes that have been renamed. RENAME ------ If a user mailbox is renamed with the IMSP "RENAME" command, the mailboxes file should be adjusted as appropriate. The IMSP "RENAME.BBOARD" command will adjust the bboards file and add a line to "changed" file. This allows modification of individual user's bbsubs file to be done in a lazy-evaluation style and also allows CMU-IMSP to alert the user (through an unsolicited OK) that the bboard has been renamed. The "RENAME" command will be implemented by proxy to the appropriate IMAP server. The "RENAME.BBOARD" command will require USP or an IMAP extension. DELETE ------ The IMSP "DELETE" and "DELETE.BBOARD" commands will adjust the bboards or mailboxes file as appropriate (no change is necessary for deleting a replication site). When a user has a subscription entry that refers to a non-existent bboard (that doesn't have an entry in the changed file) they will be informed that the bboard was deleted and the subscription entry will be removed. The "DELETE" command will be implemented by proxy to the appropriate IMAP server. The "DELETE.BBOARD" command will require USP or an IMAP extension. ACLS ---- The IMSP "SETACL", "DELETEACL", "GETACL" and "MYACL" commands only require location of the bboard on the part of the CMU-IMSP server. The rest is dependent upon implementing ACL support in either IMAP or USP. CMU-IMSP will assume that the lookup access right is always set for all bboards. LOGGING ------- CMU-IMSP will support multiple levels of logging using the standard UNIX syslog mechanism. Logging will be modular so that an alternate mechanism could be used if syslog is deemed too primitive. The "imsp.log.level" option will specify the logging level (each level includes the previous levels) as follows: 0 - only fatal errors will be sent to syslog as LOG_ERROR. 1 - warnings will be sent to syslog as LOG_WARNING. 2 - bboard administrative actions (including new bboard creations) will be sent to syslog as a LOG_NOTICE. The message will include the user, hostname, type and time of the action. 3 - every time a user logs in or out, a syslog LOG_NOTICE message will be sent. 9 - debugging messages will be sent to syslog as LOG_DEBUG. MONITORING ---------- The initial implementation will keep in mind that we will want to monitor serious IMSP errors and possibly the number of active connections to a given server. These might be made available through SNMPcon or a similar mechanism. USP FEATURES ------------ A directed move/copy function is necessary to implement the IMSP "MOVE"/"MOVE.BBOARD" command and replication. The CMU-IMSP server must be able to request available free space on a server in order to implement load-balanced creation policies. ACL support should be added to USP (or IMAP) in order to allow implementation of the IMSP ACL commands. A way of finding a system load parameter on IMAP servers should be added to support load-balanced bboard replication. Support would also be needed in order to implement the partition based CREATE feature. Also extensions to IMAP or USP commands would be needed to implement the CREATE.BBOARD, DELETE.BBOARD, RENAME.BBOARD, and ALIAS.BBOARD. INTERFACE TO DELIVERY SYSTEM ---------------------------- The delivery system is expected to use CMU-IMSP to locate the appropriate IMAP server to deliver a message. It must have "imsp.admin.subs" level access to locate the INBOX for any user. PREDEFINED OPTIONS ------------------ The following option names are reserved in this implementation. bboard.listorder [READ-WRITE] This is a list of patterns used to order the results of a "FIND BBOARDS *" or "FIND UNSEEN.BBOARDS *" command. bcc.mailbox [READ-WRITE] The name of a mailbox to APPEND blind carbon copies. date [READ-ONLY] (magic) When a user asks for the value of the date option, an RFC-822 date string should be returned with the current time. This is provided to assist small clients with unreliable clocks. delivery.hosts [READ-ONLY] This contains the list of recommended SMTP hosts for mail delivery. domain [READ-ONLY] When a user asks for the domain option, the local mail domain is returned. from [READ-ONLY] (magic) When a user asks for the value of the from option, an RFC-822 address for that user is returned. imap.default.server [READ-ONLY] This specifies the default imap server for a user. imap.new.bboard.servers [NON-VISIBLE] This specifies a list of imap servers where new bboards may be created. imap.new.mailbox.servers [NON-VISIBLE] This specifies a list of imap servers where an INBOX may be created for a new user. imap.servers [READ-ONLY] This global option contains a list of all IMAP servers managed by IMSP. imsp.admin.all [NON-VISIBLE] This is a list of users that may use any implemented IMSP features. imsp.admin.bboards [NON-VISIBLE] This is a list of users that may create, rename, delete and alias any bboards. imsp.admin.subs [NON-VISIBLE] This is a list of users allowed to view (but not change) other user's subscriptions and mailboxes. imsp.create.new.users [NON-VISIBLE] If this global option is on, the directory for a new user will be created automatically. Otherwise the system administrator must create a directory for each authorized user. imsp.create.policy [NON-VISIBLE] This is specifies the creation policy for new bboards. The option is specified as a site-defined string. imsp.default.bbsubs [NON-VISIBLE] A list of the default bboard subscriptions given to a new user. imsp.external.subs [NON-VISIBLE] If this is set, subscriptions to external mailboxes and bboards are allowed by using the {hostname}folder notation. imsp.log.level [NON-VISIBLE] This integer specifies the amount of logging to be done. imsp.options.quota [NON-VISIBLE] This is an integer specifying the maximum number of kilobytes in an options file. imsp.required.bbsubs [NON-VISIBLE] Users will not be allowed to unsubscribe to bboards in this list. imsp.share.mailboxes [NON-VISIBLE] If this global option is on, then bboard names beginning with the prefix "user.<username>." are reserved as mappings of individual user's MAILBOXES into the BBOARDS namespace. In addition, it permits users to allow other users to read their mailboxes if ACLs permit. mailbox.listorder [READ-WRITE] This is a list of patterns used to order the results of a "FIND MAILBOXES *" or a "FIND UNSEEN.MAILBOXES *" command. PROJECT PLAN ------------ Phase 1 - Basic Implementation The goal of phase 1 will be to produce an IMSP server with all basic functionality that works with a stock IMAP client. Only features which can be implemented without the USP will be done. This includes the following: Options: including GET, SET, and UNSET. FIND * (although the "SEEN" and "LAST" commands won't be implemented) Administrative levels with LOGIN SUBSCRIBE, UNSUBSCRIBE CREATE/RENAME/DELETE without replication or free-space create policy. Phase 2 - Unseen Information This will involve implementing the "SEEN" and "LAST" commands as well as modifying an IMAP server to use them. Phase 3 - Advanced features This will involve implementing the USP, "MOVE", "MOVE.BBOARD", create/rename/delete/alias for bboards, ACLs, and free-space create policy. It will probably be necessary to write our own IMAP implementation in order to make the IMAP and IMSP servers work together smoothly with these features. Phase 4 - Replicated IMSP This might be done before phase 3, as it has no dependencies on IMAP modifications. This involves adding inter-IMSP server synchronization. Phase 5 - Replicated bboards This could be done before phase 4 but depends on phase 3. Add replication support for bboards as well as load-balanced server lists returned by the FIND command.